Image processing apparatus and control method thereof

ABSTRACT

This invention makes it possible to obtain an enlarged image while suppressing noise and maintaining the sharpness of an original image only by setting a simple enlargement ratio without no special knowledge. Original image data (Ia) is enlarged in accordance with a set enlargement ratio (E) to generate an enlarged image (IA). The enlarged image (IA) is smoothed by using a smoothing filter with a size depending on the enlargement ratio (E) to generate smoothed image data (IB). Difference image data (IC) is generated by calculating the difference between the enlarged image data (IA) and the smoothed image data (IB). The generated difference image data is multiplied by an emphasis coefficient. The product is added to the enlarged image data, thereby obtaining image data (IS) that has undergone enlargement/unsharp masking.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image enlargement technique.

2. Description of the Related Art

Unsharp masking is a process of emphasizing the edge portion of animage. This designation derives from the presence of image datasmoothing included in a process of emphasizing the edge.

In unsharp masking, smoothing is executed for original image data Ia asa process target to generate a smoothed image Ib. Smoothing is a processof averaging a pixel of interest and neighboring pixels around it byusing a smoothing filter 201 as shown in FIG. 2A and changing the valueof the pixel of interest, thereby shading the image. Image data Is withimproved sharpness is generated based on the smoothed image data Ib andoriginal image data Ia. These images hold a relationship given by

Is=Ia+α(Ia−Ib)  (1)

Equation (1) will be described briefly. An edge component (i.e., ahigh-frequency component of an image) is extracted by calculating thedifference between the original image Ia and the smoothed image Ib. Theedge component is emphasized by multiplying it by an emphasiscoefficient α. Then, the emphasized edge component is added to theoriginal image Ia, thereby obtaining the image data Is with improvedsharpness.

The print resolutions of printing apparatuses are steadily risingrecently. A print resolution indicates the number of pixels per unitlength (or unit area). To print by using a printing apparatus having ahigh print resolution, the number of pixels of image data also needs toincrease. That is, a print target image needs to be large, the resultantimage data Is of equation (1) is generally enlarged as needed.

However, if the image data that has undergone unsharp masking isenlarged, the specially emphasized edge portion smoothes, resulting inlittle effect of unsharp masking.

A technique is known which executes unsharp masking by setting theemphasis coefficient α of equation (1) in accordance with the imageenlargement ratio and then enlarges the image (e.g., Japanese PatentLaid-Open No. 2003-141532). This reference also discloses another methodwhich enlarges an image first and then executes edge emphasis by usingthe emphasis coefficient α with a large value.

In the above-described technique, if an image is enlarged after edgeemphasis, the edge smoothes, resulting in noticeable image degradation.Even when edge emphasis is done after enlargement, the smoothing effectdeteriorates because the edge area of the enlarged image is large. Forthese reasons, it is difficult to emphasize the edge portion only bymaking the emphasis coefficient α large. Conversely, the processuselessly emphasizes parts other than the edge portion, and noise in theimage increases.

To cope with an increase in edge area, a method of increasing the sizeof a filter for edge detection of an enlarged image is known (e.g.,Japanese Patent Laid-Open No. 2003-274157), although this technique isnot related to unsharp masking. According to this reference, a smoothingfilter with a larger size can cope with an increase in edge area.However, parts other than the edge area are also emphasized, and noisein the image increases because of the enhanced smoothing effect.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of theabove-described problems and executes, in image enlargement, unsharpmasking for an enlarged image as much as possible instead of enlargingan image that has undergone edge emphasis by unsharp masking. Thepresent invention thus provides a technique of generating an enlargedimage while suppressing noise and maintaining the sharpness of anoriginal image by a simple operation without any special knowledge.

To solve the above-described problems, an image processing apparatus ofthe present invention has, e.g., the following arrangement.

There is provided an image processing apparatus which generates enlargedimage data by increasing the number of pixels of original image data inaccordance with an enlargement ratio, comprising:

an enlargement ratio setting unit adapted to set the enlargement ratio;

an image enlargement unit adapted to enlarge the original image data inaccordance with the set enlargement ratio;

a smoothing unit adapted to smooth the enlarged image data generated bythe image enlargement unit by using a smoothing filter having a sizecorresponding to the set enlargement ratio;

a difference image generation unit adapted to generate difference imagedata by calculating a difference between the enlarged image data andsmoothed image data obtained by the smoothing unit; and

an output image generation unit adapted to generate output enlargedimage data on the basis of the difference image data generated by thedifference image generation unit and the enlarged image data obtained bythe image enlargement unit.

According to the present invention, it is possible to obtain an enlargedimage while suppressing noise and maintaining the sharpness of anoriginal image only by setting a simple enlargement ratio without anyspecial knowledge.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram of a system according to the presentinvention;

FIGS. 2A and 2B are views showing examples of smoothing filters;

FIG. 3 is a view showing the relationship between enlargement ratios anddifference value vs. emphasis coefficient tables;

FIG. 4 is a table for calculating a filter size and an emphasiscoefficient table from an enlargement ratio;

FIG. 5 is a flowchart illustrating the process procedure ofenlargement/unsharp masking according to the first embodiment;

FIG. 6 is a view showing the correlation between an address and an imagedata scan order;

FIG. 7 is a view showing changes in GUI of an application programaccording to the embodiment;

FIG. 8 is a flowchart illustrating the main process of the applicationprogram according to the embodiment;

FIG. 9 is a flowchart illustrating the process procedure ofenlargement/unsharp masking according to the second embodiment;

FIG. 10 is a flowchart illustrating the process procedure ofenlargement/unsharp masking according to the third embodiment;

FIGS. 11A and 11B are views for explaining the contents of imageenlargement by a bi-linear method and a bi-cubic method, respectively;

FIG. 12 is a view for explaining the outline of enlargement/unsharpmasking according to the first embodiment;

FIG. 13 is a view for explaining the outline of enlargement/unsharpmasking according to the second embodiment;

FIG. 14 is a view for explaining the outline of enlargement/unsharpmasking according to the third embodiment;

FIG. 15 is a view showing a table for hardware resource determinationaccording to the fourth embodiment;

FIG. 16 is a flowchart illustrating the main process of the fourthembodiment; and

FIG. 17 is a view showing a GUI for optimum enlargement settingaccording to the fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

The embodiments of the present invention will be described below indetail with reference to the accompanying drawings. An assumption willbe described first.

Letting Ia be an original image, Ib be an image obtained by smoothingthe original image Ia, and Is be an image that has undergone unsharpmasking, equation (1) described above is obtained. The image Is obtainedby unsharp masking is enlarged (interpolated) by a bi-linear method inaccordance with an enlargement ratio E, thereby obtaining an image IS.The relationship between IS and Is is given by

IS=E(Is)=E(Ia+α(Ia−Ib))  (2)

where E(x) indicates increasing the number of horizontal pixels and thenumber of vertical pixels of an image x to E times. The smoothed imageIb is obtained by using a smoothing filter f for the original image Ia.When Ib is replaced with “f(Ia)” to explicitly indicate it, equation (2)is rewritten to

IS=E(Is)=E(Ia+α·(Ia−f(Ia)))  (2)′

With this image process, the image is smoothed by enlargement,eventually resulting in little unsharp masking effect. To cope withthis, an emphasis coefficient α needs to have a larger value, assumingenlargement. However, a larger value than necessary inevitably generatesnoise and degrades the image quality. Additionally, an inexperienceduser repeatedly executes operations of unsharp masking and operations ofenlargement to determine the optimum coefficient α and the filter sizefor smoothing. This requires a long time to obtain an image with adesired quality.

In the present invention, however, a user only needs to do an operationto execute enlargement and unsharp masking and can also obtain enlargedimage data without degradation in image quality.

Although details will be described later, the first embodiment generatesthe image IS that has undergone enlargement/unsharp masking by

IS=E(Ia)+α·(E(Ia)−F(E(Ia),E))  (3)

where F(x,E) indicates smoothing the image x by using a smoothing filterF determined by the enlargement ratio F.

FIG. 12 is a conceptual view of the flow and process of image dataaccording to equation (3).

First, the original image data Ia is enlarged in accordance with theenlargement ratio E to obtain enlarged image data IA. The enlarged imagedata IA corresponds to E(Ia) of equation (3). The enlarged image data IAis smoothed by using the smoothing filter F(,E) with a size determinedby the enlargement ratio E to obtain smoothed image data IB. Thesmoothed image data IB corresponds to F(E(Ia),E) of equation (3). Thedifferences of pixels between the enlarged image data IA and thesmoothed image data IB are calculated to obtain difference image dataIC. The difference image data IC corresponds to “E(Ia) F(E(Ia),E)” ofequation (3).

The value of each pixel of the difference image data IC is multiplied bythe emphasis coefficient α. The products are added to the values ofcorresponding pixels in the enlarged image data IA, thereby obtainingthe image IS that has undergone enlargement/unsharp masking.

The second embodiment obtains the image IS that has undergoneenlargement/unsharp masking by

IS=E(Ia)+α·(E(Ia)−E(f(Ia))  (4)

FIG. 13 is a conceptual view of the flow and process of image dataaccording to equation (4).

First, the original image data Ia is smoothed by using the averagesmoothing filter f based on, e.g., the size of the original image dataIa to obtain the smoothed image data Ib. The smoothed image data Ibcorresponds to f(Ia) of equation (4).

Each of the original image data Ia and smoothed image data Ib isenlarged in accordance with the enlargement ratio E to obtain theenlarged image data IA and enlarged smoothed image data IB. The enlargedimage data IA corresponds to E(Ia) of equation (4). The enlargedsmoothed image data IB corresponds to F(f(Ia)) of equation (4).

The differences of pixels between the enlarged image data IA and theenlarged smoothed image data IB are calculated to obtain the differenceimage data IC.

The difference image data IC corresponds to “E(Ia)−E(f(Ia))” of equation(4).

The value of each pixel of the difference image data IC is multiplied bythe emphasis coefficient α. The products are added to the values ofcorresponding pixels in the enlarged image data IA, thereby obtainingthe image IS that has undergone enlargement/unsharp masking.

The third embodiment obtains the image IS that has undergoneenlargement/unsharp masking by

IS=IA+α·E(Ia−f(Ia))  (5)

FIG. 14 is a conceptual view of the flow and process of image dataaccording to equation (5).

First, the original image data Ia is smoothed by using the averagesmoothing filter f based on, e.g., the size of the original image dataIa to obtain the smoothed image data Ib. The smoothed image data Ibcorresponds to f(Ia) of equation (5).

The differences of corresponding pixels between the original image dataIa and the smoothed image data Ib are calculated to obtain differenceimage data Ic.

Each of the original image data Ia and difference image data Ic isenlarged in accordance with the enlargement ratio E to obtain theenlarged image data IA and enlarged difference image data IC. Theenlarged image data IA corresponds to E(Ia) of equation (5). Theenlarged difference image data IC corresponds to “E(Ia−f(Ia))” ofequation (5).

The value of each pixel of the difference image data IC is multiplied bythe emphasis coefficient α. The products are added to the values ofcorresponding pixels in the enlarged image data IA, thereby obtainingthe image IS that has undergone enlargement/unsharp masking.

<Description of Apparatus Arrangement>

Common items of the embodiments will be described.

FIG. 1 shows a system configuration according to the present invention.The image processing apparatus comprises, e.g., a personal computer.Referring to FIG. 1, the apparatus includes an input device 101 such asa keyboard and a mouse, a display device 102, and a mass storage device103 such as a hard disk drive. The storage device 103 stores an OS(Operating System), application programs to be described in theembodiments, and image data files. A CPU 104 controls the entireapparatus. A ROM 105 stores a BIOS and a boot program. A RAM 106 servesas a work area of the CPU 104.

When the apparatus is powered on, the CPU 104 functions as aninformation processing apparatus by starting a process in accordancewith the boot program in the ROM 105, loading the OS from the storagedevice 103 to the RAM 106 and executing it. The user inputs anactivation instruction of an image processing application by operatingthe input device 101. The CPU 104 loads the application program from thestorage device 103 to the RAM 106 and executes it. Consequently, theapparatus functions as an image processing apparatus.

When the application is activated, and the user designates a desiredimage file in the storage device 103, the CPU 104 reads out thedesignated image file from the storage device 103 to the RAM 106 anddisplays it on the display device 102.

FIG. 7 shows a GUI displayed on the display device 102 upon reading outan image file. As shown in FIG. 7, the application displays a window 701having an image display area 702 and an optimum enlargement button 705.There are many kinds of image editing processes in addition toenlargement, although they are not illustrated.

In the example shown in FIG. 7, an enlargement target image 703 isdisplayed in the image display area 702. When the user designates theoptimum enlargement button 705 by operating the input device 101, theCPU 104 displays a dialog box 707 shown in FIG. 7 to set an enlargementratio. The dialog box includes a slider bar 707 a to set an enlargementratio, an OK button 707 b, and a cancel button 707 c. The slider bar 707a in FIG. 7 can set an enlargement ratio from 1× to 10×. The user sets adesired enlargement ratio by operating the slider bar 707 a. When theuser designates the OK button 707 b, the dialog box 707 disappears, andenlargement/unsharp masking to be described later is executed. When theuser designates the cancel button 707 c, the dialog box 707 simplydisappears to restore the state before the optimum enlargement button705 is designated. In the above-described example, the user sets anenlargement ratio by using the slider bar. Instead, the user may input anumerical value as an enlargement ratio by operating the keyboard. Thatis, an arrangement of any type is usable to set an enlargement ratio.

FIG. 8 is a flowchart illustrating the main process procedure of theapplication program. The CPU 104 executes the steps.

The CPU 104 displays the window shown in FIG. 7 in step S1 and waitsuntil the user selects an editing target image file in step S2. Toselect an image file, a dialog box for file selection is displayed. Theuser designates a target file from the input device 101. When the userselects a file, the image is displayed in the image display area 702. Instep S3, the CPU 104 waits an editing instruction from the user. If theCPU 104 determines that the user's edit instruction is not the optimumenlargement button 705, the process advances to step S4 to execute acorresponding process.

If the CPU 104 determines that the user's edit instruction is theoptimum enlargement button 705, the process advances to step S5 todisplay the dialog box 707 for enlargement ratio setting and to causethe user to set an enlargement ratio. After enlargement ratio setting,if the user designates the OK button in the dialog box 707, the processadvances to step S6 to execute an automatic optimum enlargement process.An image that has undergone the automatic optimum enlargement isdisplayed in the image display area 702 as an image 706.

A supplementary explanation of enlargement used in this embodiment willbe done. This embodiment uses a bi-linear method or a bi-cubic method asenlargement.

FIG. 11A is a view for explaining a linear interpolation method(enlargement method) used in image enlargement. To enlarge an imageincluding two pixels Xa and Xb to n times, (2n−2) interpolation pixelsare generated between the two pixels Xa and Xb. Let s be the distancefrom the pixel Xa, and t be the distance from the pixel Xb. The value ofan arbitrary pixel X is given by

X=(s×Xa+t×Xb)/2n

Such linear interpolation between two pixels is a linear method. Imagedata is two-dimensionally interpolated. That is, linear interpolation isexecuted from four neighboring pixels. A bi-linear methodtwo-dimensionally extends the linear interpolation to 2×2 pixels. As acharacteristic of the bi-linear method, the higher the enlargement ratiois, the smoother the whole image becomes. Hence, an effect similar tosmoothing can be obtained after enlargement.

FIG. 11B is a view for explaining a bi-cubic method. In this example, animage including four pixels Xa, Xb, Xc, and Xd is enlarged to n times.In the bi-cubic method, the distance between two existing pixelsadjacent in the horizontal (or vertical) direction is defined as “1”. Inobtaining the value of the pixel X to be generated for interpolation,let d be the distance between the pixel X and a neighboring existingpixel. Cubic(d) is calculated by when 0<|d|<1:

Cubic(d)=1−2|d| ² +|d| ³

when 1≦|d|<2:

Cubic(d)=4−8|d|+5|d| ² −d| ³

when 2≦|d|:

Cubic(d)=0

Cubic is calculated for the distance between the interpolation pixel Xand one of the pixels {Xa, Xb, Xc, Xd}. The sum of Cubic is theinterpolation pixel X. That is, in the example shown in FIG. 11B, theinterpolation pixel X is calculated by

X=Cubic(n+s)×Xa+Cubic(s)×Xb+Cubic(t)×Xc+Cubic(n+t)×Xd

Such pixel interpolation in a range wider than linear interpolation is abi-cubic method. Image data is two-dimensionally interpolated. That is,interpolation is executed from 16 neighboring pixels. The bi-cubicmethod readily leaves the edge portion of an image as compared to thebi-linear method. Hence, a relatively sharp impression can be obtainedeven after enlargement.

Three embodiments of the present invention will be described below. Eachembodiment has a characteristic feature in the process corresponding tostep S6 in FIG. 8. The remaining parts are common.

First Embodiment

In the first embodiment, enlargement/unsharp masking is executed inaccordance with equation (3). More specifically, the smoothing filtersize and the emphasis coefficient table are switched in accordance withthe enlargement ratio of an image. Appropriate unsharp masking isexecuted for an enlarged image. In the following description, a RAM 106stores data generated during various kinds of processes.

The sharpness process of the first embodiment is unsharp masking, asdescribed above. In unsharp masking, a smoothed image of an originalimage is generated. The differences of pixels between the smoothed imageand the original image data are calculated. Then, the values are addedto the corresponding pixel values in the original image data, thusexecuting unsharp masking.

The smoothed image is generated by using a 3×3 smoothing filter 201shown in FIG. 2A.

Ia(x,y) is the pixel value of coordinates (x,y) when the origin (0,0) isset at the upper left corner of the original image data. Ia(x,y)contains R, G, and B components (each having 8 bits). Unless otherwisespecified, pixel value calculation is done for each of R, G, and Belements. When the coordinates of the upper left corner of the imagedata is (0,0), a pixel value Ib(x,y) of the coordinates (x,y) iscalculated by

$\begin{matrix}{{{Ib}\left( {x,y} \right)} = {\left\{ {{{Ia}\left( {{x - 1},{y - 1}} \right)} + {{Ia}\left( {x,{y - 1}} \right)} + {{Ia}\left( {{x + 1},{y - 1}} \right)} + {{Ia}\left( {{x - 1},y} \right)} + {{Ia}\left( {x,y} \right)} + {{Ia}\left( {{x + 1},y} \right)} + {{Ia}\left( {{x - 1},{y + 1}} \right)} + {{Ia}\left( {x,{y + 1}} \right)} + {{Ia}\left( {{x + 1},{y + 1}} \right)}} \right\}/9}} & (6)\end{matrix}$

Next, the difference between an original image Ia and a smoothed imageIb is obtained. A difference value Ic(x,y) is given by

Ic(x,y)=Ia(x,y)−Ib(x,y)  (7)

Let α be the emphasis coefficient. A pixel value Is(x,y) after unsharpmasking is calculated by

Is(x,y)=Ia(x,y)+α×Ic(x,y)  (8)

This calculation is done for all pixels, thereby executing unsharpmasking.

A smoothed image can also be generated by switching the size of thesmoothing filter. An average value is obtained by using a 5×5 filterlike a smoothing filter 202 in FIG. 2B. A 7×7 filter and a 9×9 filterare also usable, although not illustrated. A larger filter capable ofsmoothing a wider edge portion is useful for an enlarged image. However,a large filter also emphasizes an area other than the edge portion andmakes noise in the image data noticeable.

FIG. 3 shows emphasis coefficient tables to reduce noise.

This determines an appropriate emphasis coefficient α in accordance withthe difference value Ic(x,y) obtained by equation (7). Each table inFIG. 3 has an abscissa representing the difference value and an ordinaterepresenting the emphasis coefficient α and has a curve characteristiccapable of calculating the emphasis coefficient α based on thedifference value. As a characteristic feature, the difference value islarge at an edge portion and small in an area other than the edgeportion. For this reason, when the difference value is small, a smallemphasis coefficient is adopted to prevent emphasis of parts other thanthe edge portion, thereby suppressing noise. When the filter size islarge, the difference value becomes large at a part other than the edgeportion. Hence, an appropriate emphasis coefficient table is prepared inaccordance with the filter size. In FIG. 3, the filter size increases asthe image enlargement ratio becomes high. In correspondence with this,this embodiment prepares three coefficient tables 301 to 303 withdifferent curve characteristics. As shown in FIG. 3, as the enlargementratio rises, the emphasis coefficient becomes smaller in a range wherethe difference value is small.

A table 401 in FIG. 4 shows how to change the filter size and emphasiscoefficient table in accordance with an image enlargement ratio E in thefirst embodiment. The table 401 is held in the application program.However, it may be saved in a storage device 103 separately as anenvironment file to be referred to by the application.

As shown in FIG. 4, in the range of 1.0<E<2.0 where E is the imageenlargement ratio, the filter size is 3×3, and the table 301 in FIG. 3is used as the emphasis coefficient table.

In the range of 2.0<E<5.0, the filter size is 5×5, and the table 302 inFIG. 3 is used as the emphasis coefficient table. In the range of 5.0<E,the filter size is 7×7, and the table 303 in FIG. 3 is used as theemphasis coefficient table. Even when the enlargement ratio is 5× ormore, the filter size and emphasis coefficient table can be more finelydefined. However, if the filter size becomes larger, a very long periodof time is required for smoothing. Hence, the table 303 is always usedat an enlargement ratio of 5× or more.

In the first embodiment, enlargement and unsharp masking are executed inaccordance with equation (3) described above in the above-described way.

FIG. 5 is a flowchart illustrating details of the above-describedprocess in step S6 in FIG. 8 in the first embodiment. Note that theenlargement ratio E has already been set when advancing to step S6, asdescribed above.

In step S501, the original image data Ia (image data selected in step S2in FIG. 8) is read out. In step S502, the readout image data Ia isenlarged by a bi-linear method at the set enlargement ratio E togenerate image data IA. The RAM 106 temporarily stores the generatedimage data IA. The generated enlarged image data IA corresponds to E(Ia)of equation (3). The image data may be enlarged not by a bi-linearmethod but by a bi-cubic method.

In step S503, a smoothing filter F having a size corresponding to theenlargement ratio E is determined. In step S504, the generated enlargedimage data IA is smoothed by using the smoothing filter F to obtainsmoothed image data IB. The RAM 106 temporarily stores the smoothedimage data IB, too. The generated smoothed image data IB corresponds toF(E(Ia),E)) of equation (3). In step S505, an emphasis coefficient tableis determined based on the enlargement ratio E. An area to store imagedata IS that has undergone unsharp masking is allocated in the RAM 106.

In step S506, a variable i representing a process target pixel positionis initialized to “0”. The ith pixel of image data indicates an addressvalue in raster-scanning the image data while defining the upper leftpixel as the 0th pixel, as shown in FIG. 6. Let W be the number ofpixels in the horizontal direction of the image data, and H be thenumber of pixels in the vertical direction. The total number N of pixelsof the image data is given by W×H. When the coordinates of a pixel ofinterest are represented by (x,y), i=x+y×W. Since the variable i startsfrom 0, unsharp masking is executed until the variable equals (N−1).

In step S507, the difference value between the ith pixel value of theimage data IA and that of the image data IB is calculated. Let IA(i) bethe ith pixel value of the image data IA, and IB(i) be the ith pixelvalue of the image data IB. The difference image data IC(i) is given by

IC(i)=IA(i)−IB(i)

Each pixel in the image data IA and IB has only a positive value. Thedifference pixel value of the difference image data IC can also benegative. When each pixel of the original image data is represented by 8bits, each difference pixel of the difference image data has 9 bits ormore to express a negative value.

In step S508, the emphasis coefficient α is calculated based on thedifference image data IC(i) and the emphasis coefficient table (one ofthe tables 301 to 303 in FIG. 3) determined in step S503.

In step S509, the ith pixel value of the image IS that has undergoneunsharp masking is calculated by

IS(i)=IA(i)+α×IC(i)

The calculated pixel value IS(i) is stored at the ith address positionof the area allocated in the RAM 106.

In step S510, the variable i is compared with (N−1) to determine whetherunsharp masking is executed for all pixels. If the result is NO in stepS510, the process advances to step S511 to increment the variable i by“1”, and the process from step S507 is repeated.

If it is determined that unsharp masking has ended for all pixels, theprocess advances to step S512 to output the image data IS that hasundergone unsharp masking and has been generated in the RAM 106 to animage display area 702 in FIG. 7. At this time, an area in the RAM 106that stores other intermediately generated image data, except the imagedata IS, is released so that the area is usable by the OS for otherprocesses. When a save instruction (not shown) is designated on theapplication, the image data IS generated in the RAM 106 is saved in thestorage device 103 as a file.

As described above, according to the first embodiment, even a user whohas no special knowledge can obtain image data that has undergoneunsharp masking using an optimum smoothing filter and emphasiscoefficient table only by executing a simple operation of designatingthe enlargement ratio. The unsharp masking uses a smoothing filter andemphasis coefficient table suitable for the image data IA (=E(Ia))obtained by enlarging the original image Ia. Hence, the generated imagedata that has undergone unsharp masking can have a high quality whilesuppressing noise.

Second Embodiment

In the second embodiment, unsharp masking is executed in accordance withequation (4). Both the apparatus arrangement and the UI (User Interface)are common. Details of the process in step S6 in FIG. 8 will bedescribed.

FIG. 9 is a flowchart illustrating details of the above-describedprocess in step S6 in FIG. 8 in the second embodiment. Note that anenlargement ratio E has already been set when advancing to step S6, asdescribed above.

In step S901, original image data Ia (image data selected in step S2 inFIG. 8) is read out. In step S902, smoothed image data Ib is obtained byusing a smoothing filter f corresponding to the resolution or size ofthe readout image data Ia. A RAM 106 temporarily stores the generatedsmoothed image data Ib. The generated smoothed image data Ib correspondsto f(Ia) of equation (4).

In step S903, the original image Ia is enlarged in accordance with theenlargement ratio E to obtain enlarged image data IA. The enlargement instep S903 can be done by either a bi-cubic method or a bi-linear method.In this embodiment, to leave the edge of the original image Ia,enlargement is executed by a bi-cubic method.

In step S904, the smoothed image data Ib is enlarged in accordance withthe enlargement ratio E to obtain enlarged smoothed image data IB. Theenlargement in step S904 can also be done by either a bi-cubic method ora bi-linear method. In this embodiment, since the enlargement target isa smoothed image, enlargement is executed by a bi-linear method.

The RAM 106 temporarily stores the enlarged image data IA and enlargedsmoothed image data IB. The image data IA corresponds to (E(Ia) ofequation (4). The enlarged smoothed image data IB corresponds to(f(Ia)).

The process advances to step S905 to initialize a variable irepresenting a process target pixel position to “0”. The meaning of thevariable i is the same as in the first embodiment.

In step S906, a difference value IC between the ith pixel value of theimage data IA and that of the image data IB is calculated, as in stepS507 of the first embodiment. Let IA(i) be the ith pixel value of theimage data IA

IC(i)=IA(i)−IB(i)

The process advances to step S907 to calculate the ith pixel value ofimage data IS that has undergone unsharp masking by using an emphasiscoefficient α of an emphasis coefficient filer corresponding to thesmoothing filter f of the original image Ia. The calculated pixel valueIS(i) is stored at the ith address position of the area allocated in theRAM 106.

IS(i)=IA(i)+α×IC(i)

In step S908, the variable i is compared with (N−1) to determine whetherunsharp masking is executed for all pixels. If NO In step S908, theprocess advances to step S909 to increment the variable i by “1”, andthe process from step S906 is repeated.

If it is determined that unsharp masking is ended for all pixels, theprocess advances to step S910 to output the image data IS that hasundergone unsharp masking and has been generated in the RAM 106 to animage display area 702 in FIG. 7. When a save instruction (not shown) isdesignated on the application, the image data IS generated in the RAM106 is saved in a storage device 103 as a file, as in the firstembodiment.

As described above, according to the second embodiment, the samefunctions and effects as in the first embodiment can be obtained.According to the second embodiment, smoothing using the smoothing filterf is executed for the original image data Ia before enlargement. Sincethe number of process target pixels is small, the process can speed upas compared to the first embodiment. In the second embodiment, two kindsof enlargement methods, i.e., a bi-linear method and a bi-cubic methodare adaptively used on the basis of the properties of an image, therebyfurther suppressing degradation in image quality.

Third Embodiment

In the third embodiment, unsharp masking is executed in accordance withequation (5). Both the apparatus arrangement and the UI (User Interface)are common. Details of the process in step S6 in FIG. 8 will bedescribed.

FIG. 10 is a flowchart illustrating details of the above-describedprocess in step S6 in FIG. 8 in the third embodiment. Note that anenlargement ratio E has already been set when advancing to step S6, asdescribed above.

In step S1001, original image data Ia (image data selected in step S2 inFIG. 8) is read out. In step S1002, smoothed image data Ib is obtainedby using a smoothing filter f corresponding to the resolution or size ofthe readout image data Ia. A RAM 106 temporarily stores the generatedsmoothed image data Ib. The generated smoothed image data Ib correspondsto f(Ia) of equation (5).

In step S1003, difference image data Ic between the original image dataIa and the smoothed image data Ib is calculated. The difference imagedata Ic corresponds to (Ia−f(Ia)) of equation (5).

The process advances to step S1004 to compress the difference image Icby using lossless compression coding such as ZIP and save it in astorage device 103. At this time, an area in the RAM 106 that stores thegenerated smoothed image data Ib is released.

The process advances to step S1005 to enlarge the original image data Iain accordance with the enlargement ratio E. The enlargement is executedby a bi-cubic method, as in the second embodiment.

In step S1006, the saved compressed difference image data Ic is read outand decompressed to obtain the difference image data Ic. The differenceimage data Ic is enlarged in accordance with the enlargement ratio E toobtain enlarged difference image data IC. The enlarged difference imagedata IC corresponds to E(Ia−f(Ia)) of equation (5). This enlargement isalso executed by a bi-cubic method.

In step S1007, a variable i representing a process target pixel positionis initialized to “0”. The meaning of the variable i is the same as inthe first embodiment. An area to store the image data that has undergoneunsharp masking is allocated in the RAM 106.

In step S1008, the ith pixel value of image data IS that has undergoneunsharp masking is calculated by using an emphasis coefficient α of anemphasis coefficient filer corresponding to the smoothing filter f ofthe original image Ia. The calculated pixel value IS(i) is stored at theith address position of the area allocated in the RAM 106.

IS(i)=IA(i)+α×IC(i)

In step S1009, the variable i is compared with (N−1) to determinewhether unsharp masking is executed for all pixels. If NO in step S1009,the process advances to step S1010 to increment the variable i by “1”,and the process from step S1008 is repeated.

If it is determined that unsharp masking is ended for all pixels, theprocess advances to step S1011 to output the image data IS that hasundergone unsharp masking and has been generated in the RAM 106 to animage display area 702 in FIG. 7. When a save instruction (not shown) isdesignated on the application, the image data IS generated in the RAM106 is saved in the storage device 103 as a file, as in the firstembodiment.

As described above, according to the third embodiment, the samefunctions and effects as in the second embodiment can be obtained.According to the third embodiment, the loop process in steps S1008 toS1010 is more simple than that in step S906 to S909 of the secondembodiment. Hence, the process can speed up.

Fourth Embodiment

In the above-described first to third embodiments, the number of imagedata generated by the process changes. That is, it is very effective tonot only speed up the process, as described above, but also select anoptimum processing method on the basis of the remaining capacity of aRAM 106 (storage device 103 in some cases) serving as the work area inthe processing apparatus.

In the fourth embodiment, the processing method of unsharp masking bythe above-described three equations is selected on the basis of ahardware configuration, i.e., hardware resources related to theprocessing power of an image processing apparatus. The consumptionamounts of the RAM 106, process times, and image qualities of the threekinds of unsharp masking processing methods are listed below in threesteps.

Equation (3) (first embodiment): high memory utilization, long processtime, and high image quality. Equation (4) (second embodiment): mediummemory utilization, short process time, and medium image quality.Equation (5) (third embodiment): low memory utilization, medium processtime, and low image quality.

The quality of a finally obtained image that has undergoneenlargement/unsharp masking is highest in the process by equation (3),and equations (4) and (5) follow in this order. The hardware resourcesinclude the usable memory size of the RAM 106 and the processing powerof a CPU 104.

Not the total capacity but the usable memory size of the RAM 106 isincluded in the hardware resources because a PC can normally execute aplurality of applications, and the free space of the memory changesdepending on the types and number of applications that are beingexecuted. However, it may be the total capacity of the RAM 106 in anenvironment incapable of executing many unspecified applications.

The processing power of the CPU 104 is determined by the type of the CPUand its operation clock.

Information about the free space of the RAM 106 and the processing powerof the CPU 104 can easily be acquired by using, e.g., the function callof the OS (Operating System), and a detailed description thereof will beomitted there.

In the fourth embodiment, the usable memory size of the RAM 106 iscompared with two preset threshold values to determine the level of theusable memory size, i.e., large, medium, or small.

The processing power of the CPU 104 is also compared with two presetthreshold values to determine the level of the processing power of theCPU 104, i.e., high, medium, or low.

On the basis of the usable memory size of the RAM 106 and the processingpower of the CPU 104, the index value (large, medium, or small) of thehardware resources of the apparatus that executes an application in thefourth embodiment is determined.

FIG. 15 shows a table for hardware resource determination.

As shown in FIG. 15, when both the processing power of the CPU and theusable memory size are medium or more, the hardware resources aredetermined to be large. If the usable memory size is small, the hardwareresources are determined to be small. Otherwise, the hardware resourcesare determined to be medium.

When the hardware resources are large, enlargement/unsharp masking isexecuted in accordance with equation (3), i.e., the first embodiment.When the hardware resources are medium, enlargement/unsharp masking isexecuted in accordance with equation (4), i.e., the second embodiment.When the hardware resources are small, enlargement/unsharp masking isexecuted in accordance with equation (5), i.e., the third embodiment.

FIG. 16 illustrates an example of the main process of the fourthembodiment. The main process in FIG. 16 replaces FIG. 8.

The CPU 104 displays the window shown in FIG. 7 in step S1601 and waitsuntil the user selects an editing target image file in step S1602. Toselect an image file, a dialog box for file selection is displayed. Theuser designates a target file from an input device 101. When the userselects a file, the image is displayed in an image display area 702. Instep S1603, the CPU 104 waits an editing instruction from the user. Ifthe CPU 104 determines that the user's edit instruction is not anoptimum enlargement button 705, the process advances to step S1604 toexecute a corresponding process.

If the CPU 104 determines that the user's edit instruction is theoptimum enlargement button 705, the process advances to step S1605 todisplay a dialog box 707 for enlargement ratio setting and cause theuser to set an enlargement ratio. After enlargement ratio setting, ifthe user designates the OK button in the dialog box 707, the processadvances to step S1606. In step S1606, the CPU 104 analyzes theprocessing power of its own and the usable memory size in the RAM 106and obtains the index value of the hardware resources by looking up thetable in FIG. 15. In step S1607, the CPU 104 determines the obtainedindex value.

If the index value of the hardware resources is “large”, the processadvances to step S1608 to execute a process in accordance with equation(3), i.e., the flowchart in FIG. 5. If the index value of the hardwareresources is “medium”, the process advances to step S1609 to execute aprocess in accordance with equation (4), i.e., the flowchart in FIG. 9.If the index value of the hardware resources is “small”, the processadvances to step S1610 to execute a process in accordance with equation(5), i.e., the flowchart in FIG. 10.

As described above, according to the fourth embodiment, it is possibleto execute enlargement/unsharp masking adaptively in accordance with oneof equations (3) to (5) on the basis of the hardware resources of theapparatus that executes the image editing application of the fourthembodiment.

Fifth Embodiment

In the fourth embodiment, enlargement/unsharp masking is executedadaptively in accordance with one of equations (3) to (5) on the basisof the hardware resources of the apparatus.

However, a user who owns an apparatus with poor hardware resources maydesire to do enlargement/unsharp masking to ensure the highest imagequality at the expense of time.

As shown in FIG. 17, an optimum enlargement process setting menu may bedisplayed to cause the user to select “auto” or “manual”. Radio buttonsare prepared, as shown in FIG. 17. The user explicitly selects one of“auto” and “manual”. When the user sets “auto”, the process of thefourth embodiment is done.

When the user selects “manual”, he/she manipulates the slider bar toexplicitly set one of the processes of the first to third embodiments.

The setting contents are saved in, e.g., an environment file to reflectthem on the next editing.

With the above-described process, the user can select a process from thefirst to fourth embodiments, resulting in an increase in convenience forthe user.

Other Embodiments

The four embodiments of the present invention have been described above.As described in the embodiments, the present invention can beimplemented by a computer program. Hence, the present invention alsoincorporates such a computer program.

The present invention is also applicable to a system including aplurality of devices (e.g., host computer, interface device, reader, andprinter) or an apparatus (e.g., copying machine or facsimile apparatus)including a single device.

Normally, a computer program is stored in a computer-readable storagemedium such as a CD-ROM. The computer program becomes executable whenthe storage medium is set in the read device (CD-ROM drive) of acomputer and copied or installed in the system. Hence, the presentinvention also incorporates such a computer-readable storage medium.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2006-207168, filed Jul. 28, 2006, which is hereby incorporated byreference herein in its entirety.

1. An image processing apparatus which generates enlarged image data byincreasing the number of pixels of original image data in accordancewith an enlargement ratio, characterized by comprising: an enlargementratio setting unit adapted to set the enlargement ratio; an imageenlargement unit adapted to enlarge the original image data inaccordance with the set enlargement ratio; a smoothing unit adapted tosmooth the enlarged image data generated by said image enlargement unitby using a smoothing filter having a size corresponding to the setenlargement ratio; a difference image generation unit adapted togenerate difference image data by calculating a difference between theenlarged image data and smoothed image data obtained by said smoothingunit; and an output image generation unit adapted to generate outputenlarged image data on the basis of the difference image data generatedby said difference image generation unit and the enlarged image dataobtained by said image enlargement unit.
 2. The apparatus according toclaim 1, characterized in that said output image generation unitcomprises: a plurality of emphasis coefficient tables each representinga relational curve between a value of a difference pixel and an emphasiscoefficient, the tables having relational curves with different shapes;a determination unit adapted to determine one of the plurality ofemphasis coefficient tables on the basis of the enlargement ratio set bysaid enlargement ratio setting unit; and an addition unit adapted tomultiply a value of a pixel of interest of the difference image datagenerated by said difference image generation unit by an emphasiscoefficient specified by the determined emphasis coefficient table andthe value of the pixel of interest and add the product to acorresponding pixel of the enlarged image data generated by said imageenlargement unit, and outputs the sum from said addition unit as theoutput enlarged image data.
 3. A control method of an image processingapparatus which generates enlarged image data by increasing the numberof pixels of original image data in accordance with an enlargementratio, characterized by comprising the steps of: setting the enlargementratio; enlarging the original image data in accordance with the setenlargement ratio; smoothing the enlarged image data generated in theimage enlarging step by using a smoothing filter having a sizecorresponding to the set enlargement ratio; generating difference imagedata by calculating a difference between the enlarged image data andsmoothed image data obtained in the smoothing step; and generatingoutput enlarged image data on the basis of the difference image datagenerated in the difference image generating step and the enlarged imagedata obtained in the image enlarging step.
 4. The method according toclaim 3, characterized in that the output image generating stepcomprises: a plurality of emphasis coefficient tables each representinga relational curve between a value of a difference pixel and an emphasiscoefficient, the tables having relational curves with different shapes;determining one of the plurality of emphasis coefficient tables on thebasis of the enlargement ratio set in the enlargement ratio settingstep; and multiplying a value of a pixel of interest of the differenceimage data generated in the difference image generating step by anemphasis coefficient specified by the determined emphasis coefficienttable and the value of the pixel of interest and adding the product to acorresponding pixel of the enlarged image data generated in the imageenlarging step, and in the output image generating step, the sum fromthe adding step is output as the output enlarged image data.
 5. Acomputer program stored on a computer-readable storage medium that bybeing read in and executed by a computer functions as the methodaccording to claim
 3. 6. A computer-readable storage medium storing acomputer program of claim
 5. 7. An image processing apparatus whichgenerates enlarged image data by increasing the number of pixels oforiginal image data in accordance with an enlargement ratio,characterized by comprising: an enlargement ratio setting unit adaptedto set the enlargement ratio; a first image enlargement unit adapted toenlarge the original image data in accordance with the set enlargementratio; a smoothing unit adapted to smooth the original image data byusing a preset smoothing filter; a second image enlargement unit adaptedto enlarge smoothed image data obtained by said smoothing unit inaccordance with the set enlargement ratio; a difference image generationunit adapted to generate difference image data by calculating adifference between the enlarged image data obtained by said first imageenlargement unit and enlarged smoothed image data obtained by saidsecond image enlargement unit; and an output image generation unitadapted to generate output enlarged image data on the basis of thedifference image data generated by said difference image generation unitand the enlarged image data obtained by said first image enlargementunit.
 8. The apparatus according to claim 7, characterized in that saidoutput image generation unit obtains an emphasis coefficientcorresponding to a pixel of interest of the difference image datagenerated by said difference image generation unit by using a presetemphasis coefficient table, multiplies a value of the pixel of interestby the emphasis coefficient, adds the product to a corresponding pixelof the enlarged image data generated by said first image enlargementunit, and outputs the sum as the output enlarged image data.
 9. Theapparatus according to claim 7, characterized in that said first imageenlargement unit executes enlargement based on a bi-cubic method, andsaid second image enlargement unit executes enlargement based on abi-linear method.
 10. A control method of an image processing apparatuswhich generates enlarged image data by increasing the number of pixelsof original image data in accordance with an enlargement ratio,characterized by comprising: the enlargement ratio setting step ofsetting the enlargement ratio; the first image enlarging step ofenlarging the original image data in accordance with the set enlargementratio; the smoothing step of smoothing the original image data by usinga preset smoothing filter; the second image enlarging step of enlargingsmoothed image data obtained in the smoothing step in accordance withthe set enlargement ratio; the difference image generating step ofgenerating difference image data by calculating a difference between theenlarged image data obtained in the first image enlarging step andenlarged smoothed image data obtained in the second image enlargingstep; and the output image generating step of generating output enlargedimage data on the basis of the difference image data generated in thedifference image generating step and the enlarged image data obtained inthe first image enlarging step.
 11. The method according to claim 10,characterized in that in the output image generating step, an emphasiscoefficient corresponding to a pixel of interest of the difference imagedata generated in the difference image generating step is obtained byusing a preset emphasis coefficient table, a value of the pixel ofinterest is multiplied by the emphasis coefficient, the product is addedto a corresponding pixel of the enlarged image data generated in thefirst image enlarging step, and the sum is output as the output enlargedimage data.
 12. The method according to claim 10, characterized in thatin the first image enlarging step, enlargement is executed on the basisof a bi-cubic method, and in the second image enlarging step,enlargement is executed on the basis of a bi-linear method.
 13. Acomputer program stored on a computer-readable storage medium that bybeing read in and executed by a computer functions as the methodaccording to claim
 10. 14. A computer-readable storage medium storing acomputer program of claim
 13. 15. An image processing apparatus whichgenerates enlarged image data by increasing the number of pixels oforiginal image data in accordance with an enlargement ratio,characterized by comprising: an enlargement ratio setting unit adaptedto set the enlargement ratio; an image enlargement unit adapted toenlarge the original image data in accordance with the set enlargementratio; a smoothing unit adapted to smooth the original image data byusing a preset smoothing filter; a difference image generation unitadapted to generate difference image data by calculating a differencebetween the original image data and smoothed image data obtained by saidsmoothing unit; a difference image enlargement unit adapted to enlargethe difference image data obtained by said difference image generationunit in accordance with the set enlargement ratio; and an output imagegeneration unit adapted to generate output enlarged image data on thebasis of the enlarged image data obtained by said image enlargement unitand enlarged difference image data obtained by said difference imageenlargement unit.
 16. The apparatus according to claim 15, characterizedin that said output image generation unit obtains an emphasiscoefficient corresponding to a pixel of interest of the difference imagedata generated by said difference image enlargement unit by using apreset emphasis coefficient table, multiplies a value of the pixel ofinterest by the emphasis coefficient, adds the product to acorresponding pixel of the enlarged image data generated by said imageenlargement unit, and outputs the sum as the output enlarged image data.17. A control method of an image processing apparatus which generatesenlarged image data by increasing the number of pixels of original imagedata in accordance with an enlargement ratio, characterized bycomprising the steps of: setting the enlargement ratio; enlarging theoriginal image data in accordance with the set enlargement ratio;smoothing the original image data by using a preset smoothing filter;generating difference image data by calculating a difference between theoriginal image data and smoothed image data obtained in the smoothingstep; enlarging the difference image data obtained in the differenceimage generating step in accordance with the set enlargement ratio; andgenerating output enlarged image data on the basis of the enlarged imagedata obtained in the image enlarging step and enlarged difference imagedata obtained in the difference image enlarging step.
 18. The methodaccording to claim 17, characterized in that in the output imagegenerating step, an emphasis coefficient corresponding to a pixel ofinterest of the difference image data generated in the difference imageenlarging step is obtained by using a preset emphasis coefficient table,a value of the pixel of interest is multiplied by the emphasiscoefficient, the product is added to a corresponding pixel of theenlarged image data generated in the image enlarging step, and the sumis output as the output enlarged image data.
 19. A computer programstored on a computer-readable storage medium that by being read in andexecuted by a computer functions as the method according to claim 17.20. A computer-readable storage medium storing a computer program ofclaim
 19. 21. An image processing apparatus which generates enlargedimage data by increasing the number of pixels of original image data inaccordance with an enlargement ratio, characterized by comprising: anenlargement ratio setting unit adapted to set the enlargement ratio; animage enlargement unit adapted to enlarge given image data in accordancewith the set enlargement ratio; a smoothing unit adapted to smooth thegiven image data by using a set smoothing filter; a determination unitadapted to determine a level of a processing power indicated by hardwareresources of the apparatus, i.e., a preset first level, second level, orthird level; and a control unit adapted to control said imageenlargement unit and said smoothing unit to enlarge the original imagedata in accordance with the enlargement ratio set by said enlargementratio setting unit on the basis of a determination result of saiddetermination unit, wherein letting Ia be the original image data E(x)be image data obtained when said image enlargement unit enlarges imagedata x, f(x) be image data obtained when said smoothing unit smoothesthe image data x, be a difference between pixels at the same position oftwo image data, and + be a sum, α be an emphasis coefficient, and IS beenlarged image data to be obtained, when said determination unitdetermines that the processing power belong to the third level, saidcontrol unit executes a process according toIS=E(Ia)+α·(E(Ia)−f(E(Ia)) when said determination unit determines thatthe processing power belong to the second level, said control unitexecutes a process according toIS=E(Ia)+α·(E(Ia)−E(f(Ia)), and when said determination unit determinesthat the processing power belong to the first level, said control unitexecutes a process according toIS=E(Ia)+α·E(Ia−f(Ia)).
 22. The apparatus according to claim 21,characterized in that when said determination unit determines that theprocessing power belong to the third level, said smoothing unit executessmoothing by using a smoothing filter having a size depending on theenlargement ratio.
 23. A control method of an image processing apparatuswhich generates enlarged image data by increasing the number of pixelsof original image data in accordance with an enlargement ratio,characterized by comprising the steps of: setting the enlargement ratio;enlarging given image data in accordance with the set enlargement ratio;smoothing the given image data by using a set smoothing filter;determining a level of a processing power indicated by hardwareresources of the apparatus, i.e., a preset first level, second level, orthird level; and controlling the image enlarging step and the smoothingstep to enlarge the original image data in accordance with theenlargement ratio set in the enlargement ratio setting step on the basisof a determination result in the determining step, wherein letting Ia bethe original image data E(x) be image data obtained when the imageenlargement step enlarges image data x, f(x) be image data obtained whenthe smoothing step smoothes the image data x, be a difference betweenpixels at the same position of two image data, and + be a sum, α be anemphasis coefficient, and IS be enlarged image data to be obtained, whenit is determined in the determining step that the processing powerbelong to the third level, a process is executed in the control stepaccording toIS=E(Ia)+α·(E(Ia)−f(E(Ia)) when it is determined in the determining stepthat the processing power belong to the second level, a process isexecuted in the control step according toIS=E(Ia)+α·(E(Ia)−E(f(Ia)), and when it is determined in the determiningstep that the processing power belong to the first level, a process isexecuted in the control step according toIS=E(Ia)+α·(Ia−f(Ia)).
 24. A computer program stored on acomputer-readable storage medium that by being read in and executed by acomputer functions as the method according to claim
 23. 25. Acomputer-readable storage medium storing a computer program of claim 24.